Skip to content

[DCN] Add tests for cinderBackups#3963

Draft
gais-ameer-rh wants to merge 1 commit into
openstack-k8s-operators:mainfrom
gais-ameer-rh:OSPRH-28343
Draft

[DCN] Add tests for cinderBackups#3963
gais-ameer-rh wants to merge 1 commit into
openstack-k8s-operators:mainfrom
gais-ameer-rh:OSPRH-28343

Conversation

@gais-ameer-rh

@gais-ameer-rh gais-ameer-rh commented May 26, 2026

Copy link
Copy Markdown
Contributor

spec.cinder.template.cinderBackup (singluar) in DCN DT is
replaced with cinderBackups (plural) to deploy multiple cinder backups per edge sites.
Invoking hooks/playbooks/cinder_backups.yaml playbook to validates the behaviour of cinderBackups in DCN scenario.

The playbook tests different scenarios of cinder backup creation
and restoring the backups across availability zones.

Jira: [OSPRH-28343]
Signed-off-by: Gais Ameer gameer@redhat.com

@openshift-ci

openshift-ci Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci

openshift-ci Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign bshewale for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/99ff33f1bbf6461ab44e9a849ef1445c

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 35m 00s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 28m 05s
cifmw-crc-podified-edpm-baremetal RETRY_LIMIT Host unreachable in 1h 14m 55s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 08m 28s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 43s
cifmw-pod-pre-commit FAILURE in 9m 25s

Comment thread playbooks/dcn.yml Outdated
Comment thread playbooks/dcn.yml Outdated
Comment thread playbooks/dcn.yml Outdated
ansible-playbook
{{ playbook_dir }}/../hooks/playbooks/dz_storage_cinder_backups.yaml
-i {{ inventory_file }}
-e cifmw_openshift_namespace={{ cifmw_openshift_namespace | default('openstack') }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The invocation passes cifmw_openshift_namespace but the target playbook reads cifmw_openstack_namespace; different variables.

Comment thread playbooks/dcn.yml Outdated
Comment thread playbooks/dcn.yml Outdated
@gais-ameer-rh gais-ameer-rh force-pushed the OSPRH-28343 branch 4 times, most recently from 3f97050 to 10b6657 Compare June 4, 2026 15:09
Comment thread playbooks/dcn.yml Outdated
extra_args:
ANSIBLE_LOG_PATH: "{{ cifmw_basedir }}/logs/cinder_backups_test.log"
script: |
ansible-playbook -i {{ inventory_file }} {{ playbook_dir }}/../hooks/playbooks/cinder_backups.yaml

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does {{ playbook_dir }} return when this is run?

Are you sure the hooks directory is above it? ({{ playbook_dir }}/../hooks/)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{{ playbook_dir }} should be returning absolute path of the directory containing the playbook that is currently being executed

@gais-ameer-rh gais-ameer-rh Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, {{ playbook_dir }} looks fragile and not future proof.
I will update the path to more explicit {{ cifmw_repo }}/hooks/playbooks/cinder_backups.yaml

# Not implemented in this example
cinderBackups:
{% for _ceph in _ceph_vars_list %}
{% if _ceph.cifmw_ceph_client_cluster != _az_to_scaledown %}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you're deploying cinderBackup on two AZs so that you don't have to scale it down?

Is the test code running on the same two AZs.

I think that's OK. You could come back and scaling down cinder backup in a future patch.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The template has a conditional check if _ceph.cifmw_ceph_client_cluster != _az_to_scaledown to determine which AZs are to be included in the generated cinderBackups configuration.

During the default deploy loop (without scaledown), _az_to_scaledown is set empty (roles/ci_dcn_site/defaults/main.yml:29), so all the AZs in _ceph_vars_list pass the check and are included in the cinderBackups rendered.

For the optional scaledown task, user provides cifmw_ci_dcn_site_scaledown_az, and _az_to_scaledown is set to this value (playbooks/dcn.yml:85), causing the template to exclude this specific AZ from rendered configuration for scaledown task, triggering controlplane update to remove the backup service of that AZ.

It would be good to test the backup creation and restoration post scaledown task. I believe you were suggesting the same. :)

Comment thread hooks/playbooks/cinder_backups.yaml Outdated
openstackclient
openstack volume show vol-az1-backup-az2-restore-az2 -f value -c availability_zone
register: vol_az1_backup_az2_restore_az2_zone
failed_when: "'az2' not in vol_az1_backup_az2_restore_az2_zone.stdout" No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix this white space issue

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/c95b43a761e543b6b90fd5f5a96dd9a8

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 30m 46s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 26m 28s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 35m 25s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 18m 21s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 00s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 59s
cifmw-pod-pre-commit FAILURE in 8m 42s
✔️ cifmw-molecule-ci_dcn_site SUCCESS in 2m 42s

…s template

  spec.cinder.template.cinderBackup (singluar) in DCN DT is
  replaced with cinderBackups (plural) to deploy multiple
  cinder backups per edge sites

  Add cinderBackups block to roles/ci_dcn_site/templates/service-values.yaml.j2
  to enable per-AZ Ceph backup configuration on the OpenStackControlPlane CR.
  The block iterates over all active Ceph AZs, skipping any scaled-down site as required.
  Set cinderBackup.replicas to 0 in service-values.yaml.j2 as cinderBackup is deprecated

  Invoke hooks/playbooks/cinder_backups.yaml playbook to validates the
  behaviour of cinderBackups in DCN scenario.
  The playbook tests different scenarios of cinder backup creation
  and restoring the backups across availability zones.

Jira: OSPRH-28343
Signed-off-by: Gais Ameer <gameer@redhat.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants